Transfer of learning in backpropagation and in related neural network models

Given the range of neural network paradigms available at the moment, we might ask why anyone would still want to use backpropagation. An important argument for using this learning algorithm seems to be its popularity. Backpropagation has become one of the standard technologies in connectionist modelling. Although it was invented by Werbos in 1974, it has only been with the publication of the so called PDP volumes in 1986, that it has been applied to a variety of cognitive modelling tasks (McClelland and Rumelhart, 1986). Since then, backpropagation has served an important purpose as a ’generic’ and easy-to-understand error-correction learning mechanism. There are, however, strong limits to its suitability to modelling human learning and memory as we shall argue and demonstrate in this paper. After evaluating some of these limits, as well as some of the advantages, we present a number of simulations investigating interference and transfer of human learning. It will be shown that backpropagation shows hypertransfer: learning an interfering list B may actually improve performance on the original list A in some circumstances, including conditions where human subjects do not show such behaviour. We will also investigate transfer with delta-rule learning and compare the results with the human data. These networks are shown to exhibit transfer that is fully compatible with classic findings in transfer theory (Osgood, 1949). A major drawback, in general, is the very slow learning speed of the backpropagation algorithm. Several thousands of iterations (of the entire learning set) are often needed to obtain the desired behaviour. This prompted the original developers of backpropagation to include a ’momentum’ term in their learning rule (Rumelhart, Hinton, and Williams, 1986) in order to increase the learning speed. Momentum itself is not part of the actual gradient descent method that underlies the backpropagation algorithm. It is simply a heuristic that works well to speed up the convergence in most cases. Since 1986, many other authors have proposed ways to improve the algorithm’s speed (e.g., Jacobs, 1988; Kruschke and